Papers with Transformer language models
A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Models (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Large language models are often inefficient for real-world deployment due to expensive inference costs. |
| Approach: | They propose to use knowledge distillation to transfer the knowledge of the original model to a smaller, more efficient student model. |
| Outcome: | The proposed method is the best for multi-lingual and multilingual student architectures. |
Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for length extrapolation are tailored for natural language modeling, a task known to have strong recency bias. |
| Approach: | They propose two attention alignment strategies to improve T5's long-context utilization capability without fine-tuning. |
| Outcome: | The proposed methods improve the long-context utilization capability of T5 on language modeling, retrieval, multi-document question answering, and code completion tasks without any fine-tuning. |
Plausibility Processing in Transformer Language Models: Focusing on the Role of Attention Heads in GPT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using attention heads, we can explore how Transformer language models process semantic knowledge, especially regarding the plausibility of noun-verb relations. |
| Approach: | They propose to investigate how Transformer language models process semantic knowledge, especially regarding the plausibility of noun-verb relations. |
| Outcome: | The proposed model exhibits a higher degree of similarity with humans in plausibility processing compared to other Transformer language models. |
A Family of Pretrained Transformer Language Models for Russian (2024.lrec-main)
Copied to clipboard
Dmitry Zmitrovich, Aleksandr Abramov, Andrey Kalmykov, Vitaly Kadulin, Maria Tikhonova, Ekaterina Taktasheva, Danil Astafurov, Mark Baushenko, Artem Snegirev, Tatiana Shavrina, Sergei S. Markov, Vladislav Mikhailov, Alena Fenogenova
| Challenge: | Developing Transformer language models for the Russian language has received little attention . most of these LMs are developed for English, which imposes substantial constraints on the potential of the language technologies. |
| Approach: | They propose to release 13 Russian Transformer language models that span three languages . they aim to broaden the scope of NLP research directions and develop industrial solutions for the Russian language. |
| Outcome: | The proposed models are based on Russian language datasets and benchmarks. |
Suppressing Final Layer Hidden State Jumps in Transformer Pretraining (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing models exhibit only slight changes in the angular distance between the input and output hidden state vectors in the middle layers . |
| Approach: | They propose a jump-suppressing regularizer which penalizes large hidden state displacements near the final layer during pre-training. |
| Outcome: | The proposed method significantly reduces hidden state jumps in the final layer and increases model capacity. |
Condenser: a Pre-training Architecture for Dense Retrieval (2021.emnlp-main)
Copied to clipboard
| Challenge: | Prior work fine-tunes deep LMs to encode text sequences into single dense vector representations, but dense encoders require a lot of data and sophisticated techniques to train and suffer in low data situations. |
| Approach: | They propose to pre-train Transformer language models (LMs) with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation. |
| Outcome: | The proposed model improves on various text retrieval and similarity tasks by large margins over standard LMs. |
Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale (2022.tacl-1)
Copied to clipboard
| Challenge: | a novel class of Transformer language models that combine expressive power, scalability, and strong performance of Transformers and recursive syntactic compositions. |
| Approach: | They introduce Transformer Grammars, a class of Transformer language models that combine expressive power and recursive syntactic compositions. |
| Outcome: | The proposed model outperforms strong baselines on sentence-level language modeling perplexity and syntax-sensitive language evaluation metrics. |
Mechanistic Analysis Of Universality: Numerical Comparison Circuits Across Transformer Architectures (2026.acl-srw)
Copied to clipboard
| Challenge: | Mechanistic interpretability seeks to identify internal circuits within transformer language models but it is unclear whether they generalize across model families and scales. |
| Approach: | They propose to identify internal circuits within transformer language models by numerical comparisons. |
| Outcome: | The proposed model implementations are consistent across architecture and scale, the authors show . their results highlight the need for cross model comparisons to claim generalization of internal circuits. |
Identifying Linear Relational Concepts in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | a technique for finding concept directions for human-interpretable concepts is needed to find their direction in the latent space . a linear relational concept (LRC) can be used to locate concepts in hidden activations . |
| Approach: | They propose a method for finding human-interpretable concepts by inverting a linear relational embedding and using earlier object layers. |
| Outcome: | The proposed method outperforms standard probing classifiers on performance as concept classifier and ability to causally change model output. |
Polysemy Interpretation and Transformer Language Models: A Case of Korean Adverbial Postposition -(u)lo (2025.coling-main)
Copied to clipboard
| Challenge: | -(u)lo is a polysemy of the Korean adverbial postposition. |
| Approach: | They analysed attention weights of a Korean pre-trained BERT model and a fine-tuned version of -(u)lo to determine their attention weight. |
| Outcome: | The attention weights of a Korean pre-trained BERT model and a fine-tuned version show a general reduction in attention weighting and changes in the lexico-phrasal information used depending on the specific function of -(u)lo. |
Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | Existing methods for quantization-aware training and quantization for learning have limitations in dealing with accumulative quantization errors. |
| Approach: | They propose a method that minimizes accumulative quantization errors and outperforms baselines by distilling knowledge from a fine-tuned teacher network. |
| Outcome: | The proposed method minimizes accumulative quantization errors and outperforms baselines on the XGLUE benchmark. |
Modeling Event Plausibility with Consistent Conceptual Abstraction (2021.naacl-main)
Copied to clipboard
| Challenge: | Understanding natural language requires common sense, one aspect of which is the ability to discern the plausibility of events. |
| Approach: | They propose a method of forcing model consistency that improves correlation with human plausibility judgements. |
| Outcome: | The proposed method improves correlation with human plausibility judgements. |
Quadapter: Adapter for GPT-2 Quantization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Quantization-aware training (QAT) fine-tunes model parameters in response to the numerical error arising from quantization. |
| Approach: | They propose a quantization adapter that scales activations channel-wise to make activations quantization-friendly by scaling them channel- wise. |
| Outcome: | The proposed method prevents overfitting and improves quantization performance. |
Pushdown Layers: Encoding Recursive Structure in Transformer Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pushdown Layers model recursive state via stack tape that tracks estimated depths of tokens in incremental parsing . pushdown layers are drop-in replacement for standard self-attention . recursion is a key component of many aspects of intelligent behavior, authors say . |
| Approach: | They propose a self-attention layer that models recursive state via a stack tape . Pushdown Layers is a drop-in replacement for standard self- attention . |
| Outcome: | The proposed self-attention layer improves on parse tasks with a recursive-state model . it can model recursion using a stack tape that tracks estimated depths of tokens . |
SPECTER: Document-level Representation Learning using Citation-informed Transformers (2020.acl-main)
Copied to clipboard
| Challenge: | Recent Transformer language models do not leverage information on inter-document relatedness, which limits their document-level representation power. |
| Approach: | They propose a method to generate document-level embeddings using citation graphs. |
| Outcome: | The proposed method outperforms baselines on document-level tasks. |
Transformer Language Models Handle Word Frequency in Prediction Head (2023.findings-acl)
Copied to clipboard
| Challenge: | Prediction head is a crucial component of Transformer language models. Despite its direct impact on prediction, its characteristics have been overlooked in previous analyses. |
| Approach: | They examine the inner workings of the prediction head, specifically the bias parameters, and quantify the effect of controlling their frequency biases on text generation. |
| Outcome: | The prediction head is a crucial component of the Transformer language models. |
Structural Guidance for Transformer Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained Transformer language models have proven remarkably successful in learning generic transferable linguistic representations without resorting to data intensive pre-training. |
| Approach: | They propose to combine a generative parsing and a structural scaffolding idea to guide the model's representation via additional structure loss that separates the incremental constituency parse. |
| Outcome: | The proposed models achieve impressive perplexity results on language modelling datasets, perform well on grammatical judgments, and provide useful linguistic representations that benefit a wide range of downstream tasks. |
How is BERT surprised? Layerwise detection of linguistic anomalies (2021.acl-long)
Copied to clipboard
| Challenge: | a number of studies have shown that transformer-based language models detect when a word is anomalous in context, but likelihood scores do not tell the cause of the anomaly. |
| Approach: | They propose to use Gaussian models for density estimation at intermediate layers of three language models to evaluate grammaticality. |
| Outcome: | The proposed method on BLiMP shows that language models employ different mechanisms to detect different types of linguistic anomalies. |
The Curious Case of Absolute Position Embeddings (2022.findings-emnlp)
Copied to clipboard
| Challenge: | In natural language, it is not absolute position that matters, but relative position . et al., 2017) language models incorporate positional encodings that encode absolute (linear) word order. |
| Approach: | They find that Transformer language models encode word order using positional information . they also find that models that use absolute position embeddings over-rely on positional data . |
| Outcome: | The results raise questions about the efficacy of APEs to model the relativity of position information. |
Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has shown that structural supervision helps English language models learn generalizations about syntactic phenomena such as subject-verb agreement. |
| Approach: | They train LSTMs, Recurrent Neural Network Grammars, Transformer language models, and Transformer-parameterized generative parsing models on Mandarin Chinese datasets. |
| Outcome: | The proposed models learn aspects of Mandarin Chinese grammar that assess syntactic and semantic relationships. |
Semantic Relations between Text Segments for Semantic Storytelling: Annotation Tool - Dataset - Evaluation (2022.lrec-1)
Copied to clipboard
| Challenge: | Semantic Storytelling is the goal of the future to generate stories based on extracted, processed, classified and annotated information from large content resources. |
| Approach: | They propose to create an automatic classifier for semantic relations between extracted text segments from different news articles. |
| Outcome: | The proposed method has high accuracy scores and is validated by a trained model. |
Evaluation of Dataset Selection for Pre-Training and Fine-Tuning Transformer Language Models for Clinical Question Answering (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing work on EHR QA models that learn to answer questions from structured data has focused on analyzing questions or mapping questions to existing NLP based information extraction models. |
| Approach: | They conduct 48 experiments on two clinical question answering datasets . they use open-domain and domain-specific corpora to fine-tune Transformer language models . |
| Outcome: | The proposed models can learn to answer questions from unstructured notes with accuracies up to 90% on open-domain and domain-specific corpora. |
Improving Large-scale Language Models and Resources for Filipino (2022.lrec-1)
Copied to clipboard
| Challenge: | a new large-scale pretraining corpus for Filipino improves existing resources for low-resource languages . a large dataset is too small and too narrow to create robust models that perform well in modern NLP. |
| Approach: | They propose a large-scale pretraining corpus for Filipino and a new RoBERTa pretraining technique to supplant existing models trained with small corpora. |
| Outcome: | The proposed model improves on existing models for the low-resource Filipino language . the model gains 4.47% test accuracy across three classification tasks with varying difficulty . |
RoBERTuito: a pre-trained language model for social media text in Spanish (2022.lrec-1)
Copied to clipboard
| Challenge: | Pre-trained language models have been used in many natural language processing tasks . some domain-specific models have shown to improve performance in some domains . however, for languages other than English, such models are not widely available . |
| Approach: | They present a pre-trained language model for user-generated text in Spanish . it is based on 500 million tweets and has some cross-lingual abilities . |
| Outcome: | The model outperforms models trained on over 500 million tweets on a benchmark in spanish and english. |
Exploring the Learning Capabilities of Language Models using LEVERWORLDS (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models of stochastic learning involve learning general structure rules and specific properties of the instance. |
| Approach: | They propose a framework that allows the generation of physics-inspired worlds that follow a similar generative process with different distributions and their instances can be expressed in natural language. |
| Outcome: | The proposed framework allows the generation of physics-inspired worlds that follow a similar generative process with different distributions and their instances can be expressed in natural language. |
GiLT: Augmenting Transformer Language Models with Dependency Graphs (2026.acl-long)
Copied to clipboard
| Challenge: | Recent work focuses on syntactic tree structures of languages, in particular constituency tree structures. |
| Approach: | They propose a Graph-Infused Layers Transformer Language Model which leverages dependency graphs to augment Transformer language models. |
| Outcome: | The proposed model achieves better syntactic generalization while maintaining competitive perplexity compared with baseline models. |